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PREFACE 



The 1990 amendments to the Carl D. Perkins Vocational and Applied Technology Education 
Act (Perkins II) mandate significant changes in federal vocational education policy. Among 
these is the introduction of an outcome-based accountability system. The Perkins II model of 
accountability gives states moderate flexibility to customize standards and measures of per- 
formance, assigns the primary responsibility for program improvement to local programs, 
and attaches no financial rewards or sanctions to performance. 

This study, conducted for the National Center for Research in Vocational Education 
(NCRVE), University of California, Berkeley, examined vocational education accountability 
at the local level to provide information that might improve the implementation of Perkins II 
provisions. The study found that local accountability systems already exist in many voca- 
tional programs and that they can be tools for program improvement. While these systems 
should easily accommodate many of the responsibilities envisioned by Perkins II, they may 
have difficulty with others. The authors suggest ways states and local programs can improve 
the functioning of local accountability systems, and they recommend changes in federal vo- 
cational education policy to further the goals of program improvement that motivated 
Perkins II. 

This study should be of interest to local practitioners who have to adapt to the new account- 
ability requirements and to state policymakers who are charged with implementing them. It 
should also be of interest to federal policymakers who will be called upon to reauthorize fed- 
eral vocational education legislation in 1995. 
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SUMMARY 



One of the significant changes in vocational education brought about by the Carl D, Perkins 
Vocational and Applied Technology Education Act (Perkins II) was the introduction of out- 
come-based accountability. Perkins II requires states to adopt systems of core standards and 
measures of performance for judging the success of local programs and guiding program im- 
provement.^ The primary responsibility for program improvement is assigned to local pro- 
grams, with states playing an oversight and technical assistance role. The success of this in- 
novation depends upon the relevance of standards and measures to local policymaking, the 
existing incentives and pressures under which local programs operate, and the support states 
provide for local improvement. 

Unfortunately, little is known about the local program context and the likely interplay be- 
tween local conditions and Perkins II accountability mandates. The study reported here ex- 
plores accountability from the local perspective to provide insights into the ways in which lo- 
cal programs define their goals, interact with their constituents, use information, and make 
program-related decisions. It also attempts to define the role states could play in support of 
these functions. 

Interviews were conducted with local program administrators, staff, students, and e>:*ployers 
in secondary and postsecondary vocational education programs in five states. The interviews 
focused on accountability, program improvement, the relationship between the programs and 
their constituents, and the possible effects of the Perkins II requirements. 

On the basis of these interviews, we developed a model of the local accountability systems 
currently operating in vocational education programs. The model illustrates how goals, mea- 
sures, feedback, influence, and reform interact to influence program practice. The model also 
offers guidance for evaluating and strengthening local accountability systems. Finally, it can 
be used as a basis for analyzing the interplay between existing local accountability and ac- 
countability as envisioned under Perkins II. 

We draw five conclusions based on our analysis of federal policies, local accountability sys- 
tems, and the interactions between them: 

1. Most vocational programs have functioning local accountability systems consistent with 
the model envisioned in Perkins II. Although program staff are unlikely to identify local 
relationships as an "accountability system," the following elements of accountability are 
present in most vocational education programs: 

• Program goals and objectives (including outcome objectives, such as program completion 
and placement). 

• Direct involvement of constituents in program review and design. 

• Program continuation contingent upon performance. 



''•In the context of Perkins 11, a measure is a description of an outcome and a standard is a target level of 
attainment on that measure. 
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2. Local accountability systems can be effective tools for program improvement if practical 
obstacles can be overcome. Effective administrators are needed to overcome practical 
constraints and to create an atmosphere in which: 

• Goals are explicit and widely shared, serving as focal points for program review and 
reform. 

• Measures are appropriate to reflect goal attainment and provide valid information 
about program status. 

• Feedback is timely and clear, permitting staff and constituents to judge program suc- 
cess. 

• Constituents' expertise is welcome and their opinions are heeded, leading to collabora- 
tion in reform efforts. 

• Compromises can be reached between diverse constituent groups with conflicting goals 
and needs, reducing tensions and conflict. 

• Contradictory pressures from state and federal policies can be reconciled. 

3. States can do much to evaluate and strengthen local accountability systems. Although 
their direct role in local program improvement is circumscribed in Perkins II, states can 
provide technical assistance to enhance local accountability systems by: 

• Providing local educational agencies with a vocabulary for talking about accountability 
and a clear understanding of how an accountability system might function, e.g., identi- 
fying the school's constituents, clarifjdng the role of outcome data and direct experience 
in forming opinions and stimulating reform, and describing ways constituents might 
influence program decisions for the better. 

• Conducting critical reviews of local accountability systems, not merely compliance 
checks of local program outcomes; such reviews might involve questioning staff, stu- 
dents, employers, and community members about their knowledge of program outcomes 
and their voice in program activities. 

• Sharing information about effective local practices observed within the state and in 
other states. 

• Promoting consortium-building among local education agencies. 

• Evaluating state regulations and procedures affecting vocational education and aligning 
state policies to support local accountability. 

4. Neither state education agencies (SEAs) nor local education agencies (LEAs) are fully 
prepared to carry out their responsibilities vis-^-vis accountability under Perkins II. It 
appears that additional expertise will be needed to meet the new challenges created by the 
amendments: 

• States will have new oversight responsibilities, including the evaluation and strength- 
ening of local accountability systems. 

• Limits on administrative funds will force states to reevaluate the nature of the services 
they provide and to become more efficient in providing those services. 

• States will have to establish, review, and adjust their systems of performance standards 
and measures. 

• Local agencies will have to align their data systems with state standards and measures. 

• Local agencies will have to learn to link program improvement activities to outcome 
measures. 
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• Local agencies will be held responsible for a much broader spectrum of program out- 
comes. 

5. Perkins II may overemphasize state-adopted outcome-based standards and measures. 
The solitary emphasis on outcomes could lead to undesirable consequences, including; 

• Increased data requirements that will strain local capacity r^nd may drive out input- 
and process-based measures that are valuable for program improvement. 

• Pressures to narrow program services, change client selection, and emphasize quantity 
over quality in order to meet extreme targets. 

We recommend three modifications to the federal policy embodied in Perkins II and its 
associated regulations: 

1. Encourage SEAs to monitor and strengthen local accountability systems, not merely 
program outcomes. Specifically, Congress should incorporate this broader conception of 
the state's role into federal policies by delineating specific support activities states should 
engage in. 

2. Increase the emphasis on, and support for, accountability-related capacity-building at the 
state and local levels. Specifically, training should be provided to increase states' ca- 
pacities to support the use of measures and standards for program improvement and the 
enhancement of local accountability systems. 

3. Broaden the emphasis in Perkins II on outcome measures to include measures useful for 
program improvement. Specifically, the scope of required measures and standards should 
be expanded to include program inputs and processes, while an appropriate balance is 
maintained between measures of outcomes and these other factors. 



ERIC 



9 



ACKNOWLEDGMENTS 



We wish to express our sincere appreciation tc the vocational educators, vocational education 
students, employers, and concerned community members in California, Florida, Michigan, 
Ohio, and Oklahoma who talked with us and shared their opinions. Although they are not 
mentioned by name, their experiences formed the basis for the local accountability model de- 
scribed in this report and for our conclusions about the role of states and local programs. We 
also wish to thank our colleagues David Finegold and Maryann Jacobi for their helpful com- 
ments on the report; their suggestions improved both the logic and the clarity of our presen- 
tation. As always, any remaining faults are our own. Finally, Donna White is to be congrat- 
ulated for negotiating the software maze to complete this manuscript* 



ix 



10 



CONTENTS 



PREFACE 

SUMMARY 

ACKNOWLEDGMENTS 

FIGURE AND TABLES . 

Section 

1 . INTRODUCTION 



Background and Purpose 1 

Summary of the Accountability Provisions in Perkins II 2 

Methods 4 

Structure of the Report 4 

2 . LOCAL ACCOUNTABILITY AND PROGRAM IMPROVEMENT 5 

Definitions: Accountability and Accountability Systems 5 

' Local Accountability in Vocational Education 6 

Model of a Local Accountability System 7 

Accountability Systems and Program Irnprovement 9 

3 . EVALUATING AND STRENGTHENING LOCAL ACCOUNTABILITY 

SYSTEMS 16 

Goals iQ 

Measures 17 

Feedback 18 

Influence 19 

Program Planning and Reform 20 

Summary 21 

4. CONCLUSIONS AND RECOMMENDATIONS 22 

Conclusions 22 

Recommendations 27 



Appendix : THE CHANGING RESPONSIBILITIES OF LOCAL AND STATE 
EDUCATION AGENCIES UNDER PERKINS II 

BIBLIOGRAPHY 



I 



I 



xi 



ERIC 



11 



FIGURE 



2.1. Model of a Local Accountability System 7 

TABLES 

1.1. State Responsibilities Under Perkins II 3 

1.2. Local Responsibilities Under Perkins II 3 



xiii 

o 1 

ERIC 



1. E^TRODUCTION 



BACKGROUND AND PURPOSE 

The Carl D. Perkins Vocational Education Act (Perkins) authorized federal funding for voca- 
tional education between 1985 and 1990. In 1990, the act was amended in significant ways, 
and its title was changed to the Carl D. F^/kins Vocational and Apphed Technology Educa- 
tion Act (Perkins II). Among the new provisions of Perkins II are requirements that states 
establish systems of outcome-based standards for, and measures of, program performance, 
and that local districts use these standards as a basis for accountability and program im- 
provement.^ These provisions were to be implemented by September 1992.2 

The changes in the act generated considerable attention among practitioners as well as re- 
searchers. During the years just prior to the deadhne for implementing standards and mea- 
sures, states devoted considerable effort to selecting indicators and designating minimum 
performance levels. The National Center for Research in Vocational Education (NCRVE) 
supported technical assistance to states (Hoachlander et al., 1992) and funded research to 
describe vocational education governance (McDonnell and Zellman, 1992) and to explore the 
responses of states and localities to key provisions of the new law (Hoachlander and Rahn, 
1992). 

The NCRVE also funded research on the broad issue of accountability in vocational education 
(Hill et al., 1992). Hill et al. examined the relevance of outcome data to vocational education 
policymaking at the local, state, and federal levels. They concluded that aggregated outcome 
data have only limited utility for federal policymaking, and that the goals of vocational edu- 
cation can be served best by strengthening the program improvement capabilities of states 
and local programs. 

Fortunately, the accountability provisions of Perkins II are consistent with these conclusions. 
The law places greater responsibility for program improvement at the local level, while limit- 
ing the role of states to oversight and technical assistance. Tlxe law also emphasizes out- 
come-based accountability as an instrument for program improvement (although it stops 
short of imposing rewards and sanctions). Variations on the outcome-based accountability 
model have shown promise in other human-resource programs (Butler, 1988; Dickinson et 
al., 1988), and expectations are high among vocational educators and pohcymakers that local 
programs will become more accountable and will use performance data for program im- 
provement. 

However, there is little empirical information about local accountability in vocational pro- 
grams on which to base these expectations. Previous studies have not examined whether lo- 
cal programs use data for program improvement or how accountable they are to their con- 
stituents. To fill this gap, we sought to explore accountability from the local perspective. 



The word standard is used in different ways in educational policymaking; it can mean an educational goal, a 
curriculum model, or a performance target. We adopt the terminology used by Congress: A measure is "a descrip- 
tion of an outcome," while a standard is "a level or rate of that outcome.' Implicit in the adoption of a measure is a 
goal to achieve the outcome it describes. 

^The appendix contains an extensive discussion of the changes in Perldna n that affect accountability. 
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Thus far, data systems have received the bulk of attention in the flurry of activity to comply 
with Perkins II. However, our initial contacts with vocational programs revealed that rela- 
tionships with local constituents affected local decisionmaking as much as or more than the 
characteristics, or even the existence, of externally imposed standards or measures. Conse- 
quently, the purposes of this research were to gain an understanding of how local account- 
ability systems function, including how data are used at the local level, to explore the role 
states could play in support of local accountability, and to examine the impact of federal leg- 
islation (i.e., Perkins II) on local accountability. This information should complement the 
picture of accountability-writ-large drawn by Hill et al. 

Answers to the questions we have addressed are particularly relevant at the present time be- 
cause statewide standards and measures must be implemented by fall 1992, and states and 
local programs will have begun to use the standards as tools for program improvement. A 
better understanding of local accountability systems and their role in program improvement 
may help state and local practitioners function more effectively. In addition, Perkins II will 
expire in 1995, and discussions about reauthorization will begin a year or more prior to that 
date. This is an appropriate time to begin thinking about revisions that might be made to 
the reauthorization to enhance the effectiveness of vocational education programs. 

SUMMARY OF THE ACCOUNTABILITY PROVISIONS IN PERKINS H 

Perkins II contains provisions that create new formal, outcome-based accountability systems 
at the state level. The act prescribes new program accountability and program improvement 
responsibilities for state and local education agencies (SEAs and LEAs). The key features of 
this legislation are summarized below; a more complete overview of changes in the roles of 
states and local providers is presented in the appendix. 

States must fulfill three requirements that relate to program accountability and improve- 
ment: They must develop state plans, evaluate the statewide efficacy of vocational programs, 
and provide technical assistance to LEAs (see Table 1.1). From the perspective of account- 
ability, the most important provisions are those that concern evaluating programs. States 
are required to adopt systems of "core standards and measures of performance" that will 
serve as the primary monitoring tool for state and local program improvement efforts. The 
measures implicitly reflect the outcomes against which programs will be judged; the stan- 
dards serve as benchmarks for successful performance. 

Afler this system of standards and measures is in place, the state will take a secondary posi- 
tion in monitoring local program performance. If local programs fail to make substantial 
progress^ toward meeting the standards for two years in a row, SEAs must work with LEAs 
to develop joint improvement plans and must provide technical assistance to help them im- 
prove their performance. In addition, states must provide technical assistance to all LEAs in 
a number of areas, including professional development for teachers, development and dis- 
semination of curricula, and assessment of programs. They are also required to help LEAs 
implement the statewide standards and measures for program assessment. 

LEAs are responsible for developing local plans, providing services, and evaluating and im- 
proving programs (see Table 1.2). Perkins II makes it clear that the primary responsibility 



^"Substantial progress" is not defined in the legislation; the determination of what constitutes adequate progress 
appears to be left to the LEA, but thi j is not clear. 
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Table 1*1 



State Responsibilities Under Perkins II 



Develop a state plan 

• Assess current and projected occupational needs and demands 

• Seek input from interested parties 

• Assure compliance with Perkins II 

Evaluate vocational programs 

• Assess ability of state's vocational education system to address state needs 

• Develop measurable objective criteria on which to base assessment 

Provide technical assistance to LEAs 

• Implement professional development activities for teachers 

• Develop curricula 

• Develop performance standards and measures for program evaluation 



for accountability rests at the local level by specifying, for the first time, that each local plan 
must include the evaluation st^'jidards that will be used to measure program success, and by 
assigning specific evaluation responsibilities to local programs. These responsibilities in- 
clude modifying the state-adopted standards and measures to suit local conditions, conduct- 
ing annual evaluations based on those standards and measures, and developing program im- 
provement plans when their programs fail to make substantial progress toward meeting the 
standards. 

Overall, Perkins II represents a significant change in accountability at the state and local 
levels. The three most significant elements are (1) defining accountability in terms of pro- 
gram improvement rather than either implicit or explicit financial rewards or penalties, (2) 
defining program success in terms of outcomes rather than inputs or processes, and (3) giving 
LEAs primary responsibility for evaluation and improvement, with SEAs in the oversight 
role. 



Develop a local plan 

• Assess local occupational needs 

• Address needs of special populations 

• Coordinate with community-based organizations 

Provide services 

• Serve special populations 

• Use funds for direct services, not administration 

Evaluate program performance and tdke actions to improve programs 

• Modify state standards and measures to make them relevant to local conditions 

• Evaluate programs annually 

• Develop a program improvement plan if performance or progress is inadequate 



Table 1^ 



Local Responsibilities Under Perkins U 




METHODS 

Because so little is known about local accountability and data systems in vocational educa- 
tion, we were more interested in theory-building than theory-testing. Therefore, we adopted 
an open-ended approach: We interviewed vocational education program staff, students, and 
local employers in California, Florida, Michigan, Ohio, and Oklahoma. The selection of 
states was based on geographic diversity and the presence of one or more strong accotzntabil- 
ity-related factors, such as a statewide occupational competency system, a set of performance 
standards, or innovative occupational assessment tools. 

Using recommendations from each state's Office of Vocational Education and from other 
RAND and NCRVE researchers familiar with state activities, we arranged to visit at least 
two local service providers in each state. Depending on the organization of the state's voca- 
tional education system, these providers were either area vocational technical schools, joint 
vocational school districts, comprehensive high schools, vocational high schools, and/or com- 
munity colleges. 

At each site, we interviewed administrators (e.g., the superintendent, the principal, the voca- 
tional coordinator, the program coordinator), instructors, students, employers, and, occa- 
sionally, parents. Our interviews focused on accountability, program improvement, the rela- 
tionship between the program and its constituents, and the Perkins II requirements. On 
each visit, we spoke with approximately 20 people individually or in small groups. We also 
collected documents describing *^ach school's goals, programs, assessments, and accomplish- 
ments. 

Our descriptions of and conclusions about local accountability systems are based on these in- 
terviews and documents. Since neither the sample of states nor the sample of schools within 
states was representative, our findings must be interpreted with caution. They portray 
conditions in the set of vocational education programs we visited, but because our sample of 
states was not random, the common accountability elements we observed may not exist else- 
where. We have no reason to believe these schools were unusual in significant ways, how- 
ever, and we think the model derived from them is appropriate to describe other vocational 
programs. Although we do not have adequate evidence to support either claim, we were 
struck by the similarities among the programs we visited. And, more important, our re- 
search convinced us that it is possible to develop viable systems of local accountability. 

STRUCTURE OF THE REPORT 

The remainder of this report is divided into three sections. Section 2 explains what we mean 
by accountability and accountability systems and presents a model that describes how local 
accountability systems operate in the districts we visited. Section 3 discusses how states 
might evaluate and strengthen local accountability systems. Section 4 presents our conclu- 
sions and makes specific recommendations for federal actions to enhance the functions of lo- 
cal accountability systems and promote the goals of Perkins II. An appendix details the 
changing responsibilities of LEAs and SEAs for accountability under Perkins II. 
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2. LOCAL ACCOUNTABILITY AND PROGRAM IMPROVEMENT 



Perkins II places the responsibility for program improvement squarely on the shoulders of lo- 
cal programs. For this policy to be effective, local programs must be able to evaluate their 
own actions and initiate changes as needed. The act itself offers little direction for program 
administrators or state vocational educators about how to achieve better programs. It only 
mandates the development of outcome-based standards and measures of acceptable perfor- 
mance to be used to judge the success of individual programs. 

This situation is roughly analogous to trying to improve the health of the population by es- 
tablishing targets for weight and blood pressure, distributing scales and blood pressure 
gauges, but not providing any advice about diet or cardiovascular fitness. Would such a pol- 
icy promote health and fitness? That depends on the health-improvement knowledge and 
skills people possess and the assistance they receive to build such skills. Similarly, the im- 
pact of the Perkins performance standards and measures depends on the existing capacity 
for program improvement and on the support local programs receive to help them build this 
capacity. 

We investigated whether local programs are adequate to carry out their program improve- 
ment role effectively. Because program improvement necessarily entails modifying or delet- 
ing ongoing programs and instituting new ones, we asked program administrators to describe 
how they make such decisions using concrete examples. Our discussions focused on both the 
formal and informal methods they used to gather information, and where that information 
originated. We also asked them to describe state and local laws and regulations that affected 
th3ir actions, and what forms of assistance were available to g^de their decisions. 

Because student, parent, business, and community support may be important factors in 
program-related decisions, we asked representatives from these groups to describe their 
participation in particular programs and their role in the decisionmaking process. For 
example, we asked students how they decided to enroll in one program or another, and we 
asked business representatives ix) describe their relations with the school and its programs. 

We begin the section by defining the key terms accountability and accountability systems. 
We then present a conceptual model of a local accountability system, developed on the basis 
of our research and insights gained from reviewing the educational accountability literature. 
Next we describe how the system can function as a program improvement tool based on our 
interviews and observations. Finally, we consider limits to the model. In an earlier paper 
(Stecher and Hanser, 1992), we described practical limitations to local accountability, which 
are summarized at the end of this section. 

DEFINITIONS: ACCOUNTABILITY AND ACCOUNTABILITY SYSTEMS 

Throughout this report we refer to accountability, one of the most frequently used terms in 
*he current debate on educational policy. Unfortunately, accountability means different 
things to different people; Kirst (1990), for example, distinguishes six different approaches to 
accountability. In this report, accountability is taken to mean a relationship between two en- 
tities that share goals and have the capacity to influence each other's benefits. In the context 
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of vocational education, accountability represents a relationship between a service provider 
(usually a school) and its constituents^ (students, parents, employers, and the community), in 
which the following conditions ezist:^ 

1. The constituents expect the school to provide vocational education services (leading to 
employment, workforce improvement, and economic development). 

2. The school accepts these goals as part of its mission. 

3. The school derives benefits from achieving the goals. 

4. The constituents have some capacity to influence the benefits that accrue to the school. 

The nature of the benefits that accrue to schools and the influence exerted by constituents 
can vary. In the vocational education context, the most common benefits are economic re- 
wards in the form, of local tax revenues and continued or increased state and federal funding. 
The most common avenues for direct constituent influence are advisory committees and elec- 
tions. Advisory committees help set election policies; elections determine the composition of 
school boards and, in many states, the amount of local tax support. Constituents also exert 
indirect control of benefits through their participation as students or employers. In every 
case of which we are aware, state resources depend directly or indirectly on enrollment and 
placement levels, which, in turn, depend on student and employer perceptions of the quality 
of the programs and their graduates.^ 

An accountability system is a collection of formal and informal communication mechanisms, 
information exchanges, and action paths that foster shared goals and benefits. Such systems 
include ways for constituents to make their expectations clear to schools and influence the 
goals schools pursue, methods of sharing information about program performance so practi- 
tioners and constituents can monitor the quality of the programs, mechanisms for communi- 
cating opinions and judgments between clients and the school to help administrators formu- 
late and improve programs, and procedures for rewarding schools and constituents for good 
performance and sanctioning them for poor performance. An accountability system is effec- 
tive if it promotes improved services and greater benefits to constituents and schools. 

LOCAL ACCOUNTABILITY IN VOCATIONAL EDUCATION 

In the districts that we visited, we found ample evidence that nascent local accountability 
systems exist and that they can be nurtured to function as tools for program improvement. 
Two features of vocational education form the basis for these accountability systems: (1) vo- 
cational education is outcome-oriented, and (2) it involves constituents in a meaningful way 
in program planning. Local programs have an outcome orientation by design. The predomi- 
nant goals imposed by states through regulations or funding mechanisms are student com- 



use the term constituents broadly to denote all individuals and organizations in the community served by 
the school. Constituents include those who benefit directly or indirectly from the services provided. 

^Adapted from Hill and Bonan (1991). 

^Many factors determine the degree of choice employers and students have as consumers of vocationo] education 
products and services. In most cases, there are alternative sources of trained employees^ and in most urban areas 
there are multiple providers of training, so employers and students can respond to perceived quality. However, in 
many areas, choices are limited. 
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pletion and placement in jobs or further training. In many programs, instructors are directly 
responsible for placement. 

Constituent involvement is also deeply embedded in most local vocational education pro- 
grams. Local programs give voice to employers' concerns through industry and employer ad- 
visory committees that contribute to program design and goal-setting. In every LEA visited 
in this study, there was a network of active employer advisor^' committees. Also, employers 
are the judges of last resort, either hiring or not hiring graduates and thereby approving or 
not approving the results of the program. Employers and industry groups represent only one 
constituency, but they are the most widely involved in program decisions. Students influence 
programs primarily through entrance or exit decisions, either enrolling or opting for other 
activities and, once enrolled, either completing or dropping out. Where local tax levies sup- 
port vocational education, the local community also has indirect influence over program di- 
rection. Through their votes, citizens communicate their satisfaction or dissatisfaction with 
the performance of the vocational schools in broad terms. 

These conditions create an accountability relationship primarily among programs, employ- 
ers, and students, and secondarily with the broader community. In all the sites we observed, 
these elements were present. They give voice to constituents' concerns and facilitate the 
translation of evaluative judgments into actions. On the basis of our observations, we would 
expect to find local accountability systems of some kind in almost all vocational education 
programs. 

MODEL OF A LOCAL ACCOUNTABILITY SYSTEM 

From our visits, we were able to construct a general model of an idealized local accountability 
system that reflects the conditions and interactions we observed in vocational education pro- 
grams. This model is illustrated in Figure 2.1. Each of the elements is abstracted from 
conditions we observed in practice, but in no single site was the model fully realized. The 
normal flow of activity is clockwise, beginning with the administration; the cycle is continu- 
ous and ongoing. 




Figure 2.1— Model of a Local Accountability System 
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We should note at the outset that this model includes one key element that many would con- 
sider to be outside the bounds of accountability per se: program reform. Although program 
reform is separable from accountability, we include it because it is essential to the Perkins II 
view of accountabiHty. In a sense, its inclusion **completes" the accountability system by 
bringing it full circle from goal-setting, through feedback and influence, to reform. In our 
view, a model that stopped short of reflecting how data are used for program improvement 
would be far less informative and far less useful than our model for analyzing current voca- 
tional education accountability. The model v^e present includes the following components: 

• Vocational program provider. The key administrative and organizational entity may 
be a district, a school, or a unit within a school, depending on local conditions. We distin- 
guish between administrative and programmatic functions. The former include such 
things as planning, supervision, monitoring, and reform. The latter include instruction, 
assessment, and other classroom-related actions. Program staff may perform one or both 
of these functions; as instructors they are responsible for program delivery, and as mem- 
bers of department committees they often make planning decisions and perform other ad- 
ministrative functions. Administrators, by comparison, usually perform only administra- 
tive functions. 

• Goals. Organizing principles that define the purpose of the program. In vocational edu- 
cation, goals usually relate to emplojmient-related skills and outcomes, such as occupa- 
tional knowledge and job placement. However, goals also can be defined in terms of school 
capabilities (inputs), such as teacher qualifications, curriculum, materials, or accreditation 
status, and services (processes) such as instruction, tutoring, and services for special popu- 
lations. 

• Measures. Methods for quantifying status with respect to goals. Measures usually 
are numeric summaries of system performance, such as the percentage of teachers with 
industry-approved certification, the proportion of students receiving free tutoring, or the 
number of program graduates finding employment in a field related to their course of 
study. 

• Feedback. The transfer of information about program status to constituents, program 
administrators, and staff. The information can include formal reports, as well as impres- 
sions gained from direct experience observing, participating in, or reviewing programs. 

• Influence. The manner in which constituents make their opinions and expectations 
known to program staff, and the manner in which program staff communicate to con- 
stituents. Constituents have two m^or vehicles for communicating their judgments: They 
can "vote with their feet," withholding participation if they do not value the services they 
are receiving, or they can express their needs and opinions through various formal and 
informal channels. These two responses have been called "exit" and "voice" (Hirshman, 
1970). 

• Program design and reform. The actions taken by administrators to form and reform 
programs in response to measures, direct feedback about program performance, and con- 
stituent influence. Under ideal circumstances, administrators and other school staff have 
a repertoire of effective strategies they can call upon to address concerns or deficiencies in 
performance. These skills are used to redesign and reform programs in response to feed- 
back and influence. 



20 



9 



• Constituents, iembers of the community who have an interest in the services provided 
by the school o ..i the graduates if. produces. Constituents include students, their parents 
and friends, employers, trade associations, and the general business community, as well as 
the general populace who are affected by economic development. 

• State an federal governments. Funding and regulatory actions that affect vocational 
education programs. Both state and federal governments influence local accountability 
through the rules they place on operations and the level of support they provide. These 
external conditions affect many aspects of program operations, including the establish- 
ment of broad goals for vocational education, the identification of high-demand occupa- 
tions, and the choice of measures for Perkins II program review. We treat these primarily 
as fixed constraints within which local programs operate. 

ACCOUNTABILITY SYSTEMS AND PROGRAM IMPROVEMENT 

How can an accountability system help to improve local programs? As we have described it, 
accountability is a goal-directed relationship. Both parties agree on desired outcomes and 
judge and reward system performance on these criteria. The judgments and rewards drive 
program improvement toward desired goals. 

The mechanisms for change can be traced in Figure 2,1. For purposes of description, we be- 
gin with a discussion of goals and proceed through the other elements of the system. How- 
ever, it should be understood that the processes in the model are continuous. Although for- 
mal program revision may occur on an annual basis, other interactions are ongoing. There is 
constant give and take between students, instructors, administrators, and other constituents. 

Goals 

Goals, whether explicit or implicit, form the basis for program design, evaluation, and re- 
form. Their position in the center of Figure 2.1 is not coincidental. Program design proceeds 
from goals; measurement and evaluation reflect back upon goals; and program reform may 
include the reformulation of goals. This centrality of goals and objectives is reflected in much 
of the literature on program evaluation (Tyler, 1942; Popham, 1975; Rossi and Freemian, 
1989). 

In program design, the process through which goals are established and translated into ac- 
tion is not alv/ays apparent. Local vocational education goals usually are elements of an 
interlocking set of hierarchical goals that reflect local desires as well as state and federal 
mandates. This hierarchy begins with goals established at the federal level and continues 
downward through states and LEAs to individual occupational programs. 

Federal goals usually are very broad; for example, the primary purpose of Perkins II is to 
promote U.S. competitiveness in the world economy by "improving educational programs 
leading to academic and occupational skill competencies needed to work in a technologically 
advanced society."^ States are required to estabhsh plans that further refine and assign pri- 
orities to goals. At the LEA level, the state mandates are translated into more specific opera- 



^Carl D. Perkins Vocational Education Act, Sec. 2. 
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tional goals, such as identifying occupational areas in demand in the local economy, develop- 
ing training programs to prepare students for meaningful careers in those areas, and moni- 
toring the placement of graduates into occupations for which they were trained. At the pro- 
gram level, specific student learning and/or instructional objectives emerge. Instructors use 
these objectives to develop curricula, select materials, plan lessons, and write tests. In this 
complicated manner, program planning is based on goals. 

Goals also serve an important role in judging the performance of programs. In fact, goal at- 
tainment is the basis for many program evaluation models (Popham, 1975). In a local ac- 
countability system, goals are the basis for selecting measures of program performance that, 
in turn, are used to judge program success.^' In vocational programs, broad programmatic 
goals usually are translated into more incremental objectives in the areas of recruitment, en- 
rollment, retention in the program, course completion, internship completion, certification, 
placement, and retention on the job. These become the criteria against which program suc- 
cess is measured. 

Similarly, program reform comes about as the result of information and influence directed 
toward attainment of goals that are unmet or that have chgnged. Goals serve as targets for 
program redesign and reform efforts. Furthermore, if staff or constituents become convinced 
that program objectives themselves are inappropriate, goals may be reformulated. 

To serve these multiple functions, goals should be explicit, clear, measurable, and endorsed 
by constituents (Stecher and Hanser, 1992). In addition, they should be reviewed and up- 
dated regularly. Frequent review insures that the program continues to be responsive to the 
demands of t ie workplace and the needs of constituents. Finally, if goals are to be effective 
guides to program reform, there must be a system for setting priorities among them and for 
reviewing those priorities over time. 

The processes of establishing goals and translating them into actions are carried out by pro- 
gram staff and constituents. Some groups play a primary role, while others play a facilitat- 
ing role (Metfessel and Michael, 1967). In vocational education, employers, trade associa- 
tions, and other community members may make important contributions to the setting of 
program goals as well as to the setting of specific instructional objectives. 

Measures 

Measures play an important role in an accountability system because they are the chief cur- 
rency of program performance.^ When selected, collected, and reported properly, measures 
provide a common, trustworthy basis forjudging the success of programs. This is crucial for 
determining whether program reform is needed. Perkins II explicitly directs states and local 
agencies to monitor program performance in terms of outcomes, and vocational education 



^Perkins 11 mandates that states select and use a set of measures of program outcomes. We take as Implicit in 
this requirement the adoption of the underlying goal of producing these outcomes. 

^The secondary currency is direct experience with the program, which will be elaborated in the discussion of 
feedback below. 



22 



11 



programs across the country are collecting or planning to collect measures of academic-skill 
improvement, program completion, occupational-skill attainment, and job placement^ 

Most schools have access to a wealth of data that could be used to monitor progress toward 
achieving program goals and objectives. Variables that can be measured relatively easily in- 
clude recruitment, enrollment, course retention, course completion, skill attainment, certifi- 
cation, job placement, and job retention. 

One consequence of the strong emphasis on outcomes may be a drawing of attention away 
from other school factors, including inputs and process variables.® While indicators of out- 
comes are useful forjudging some aspects of overall performance, they are often incomplete 
indicators of total system performance and they offer limited suggestions for program im- 
provement (recall our earlier example of distributing scales and blood pressure gauges as a 
means of affecting weight and blood pressure). If measures are defined broadly enough (i.e., 
if they include measures of content and instructional quality) they can serve two roles in an 
accountability system: as indicators of outcomes and as guides to improvement (David, 
1987). 

To be effective elements in an accountability system, measure: must be consonant with goals 
and sufficient to describe them, of adequate technical quality, and clear and easy to under- 
stand (Stecher and Hanser, 1992). This does not mean they must be objective (rather than 
subjective) or quantitative (rather than qualitative). Even subjective judgments summarized 
through systematic consensual procedures can provide valid information about constituent 
beliefs, program impact, etc. As accountability tools, measures must provide constituents 
and program staff with an accurate, common view of goal attainment. 

Feedback 

Feedback is the mechanism through which constituents and program staff learn about pro- 
gram performance. The information they receive forms the basis for their judgments about 
program success and their subsequent actions to influence programs. Hence, feedback is an 
essential element in an accountability system. 

There are two sources of feedback: formal measures collected and reported by the program 
(such as those described above) and impressions and opinions formed through direct or indi- 
rect experience. Formal measures are shared with program staff and constituents in many 
forms, including program summary reports, evaluation reports, tabulations of outcome mea- 
sures, minutes and recommendations from advisory committee meetings, and community 
employment projections. 

Program administrators control the dissemination of formal feedback and can limit access to 
information if they choose. For example, employers, students, and parents receive most of 
their formal feedback about a school's program through official channels, in the form of writ- 
ten summaries and reports. This information varies in breadth, depth, and quality, depend- 



' As noted above, the outcome-baaed accountability requirements of Perking II have led indirectly to the adoption 
(at least implicitly) of these outcomes as program goals. 

®In the past, school quality oflen was judged in terms of inputs (e.g., facilities, teachers, equipment), and that 
model remains the basis for many school accreditation procedures. We are not suggesting a return to emphasizing 
process variables, but rather noting that a proper balance must be struck between outcome and process variables. 
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ing upon the wishes and skills of those producing it. Constituents who serve on an advisory 
committee usually have access to data more frequently than those who have no official con- 
nection with the school. 

In addition to this formal feedback, many people (including both program staff and con- 
stituents) have direct experience with the school's program. These contacts enable people to 
form their own opinions about program success. For example, students, instructors, and ad- 
ministrators have extensive personal knowledge of program operations and student perfor- 
mance. Employers who participate on advisory committees or interview program graduates 
seeking employment form impressions of the program, based on these contacts. Other con- 
stituents have indirect contact through comments or stories told by friends or associates, or 
through reports in newspapers. 

Information about goal attainment is critical if constituents and program staff are to be able 
to malce judgments and exercise their influence to bring about program improvement. Con- 
sequently, feedback that is relevant to the program's most important goals is critical in an 
accountability system. Constituents who do not have access to formal reports that summa- 
rize objective measures will form impressions on other, conceivably less valid grounds and 
will exercise their influence in a less than optimum manner. 

At first glance, it may appear that more feedback is always better. In fact, enhancing the 
quality of feedback may be more important than increasing the amount. Evaluation research 
has identified many factors that make information useful for decisionmaking, such as rele- 
vance, timeliness, interpretability, and mode of presentation (Alkin et al, 1979; Alkin et al, 
1982; Brown et al., 1978; Braskamp and Brown, 1980). Accountability depends on the com- 
munication of information about program performance, and there may be many ways to 
make such communication more accurate, meaningful, and relevant. 

Influence 

Influence is the mechanism through which constituents and program staff act to affect pro- 
gram direction. As noted above, constituents have two forms of influence over programs: 
voice and exit (Hirshman, 1970). They can voice their opinions through formal channels, 
such as advisory committees or student associations, or through informal contacts with staff. 
Such communication can be an effective way of making needs felt, influencing goals, and 
bringing about program improvements. 

Constituents also can withhold their participation or involvement if they do not feel they are 
well served by the program. In many respects, this is the ultimate constituent sanction. 
Students can leave the program or give negative recommendations to friends who are consid- 
ering enrollment. At most schools, word of mouth has a major effect on recruitment, so dis- 
satisfied students can affect future enrollment. Similarly, employers can refuse to serve on 
committees, accept interns, or hire program graduates. 

The immediate effects of these exit actions are minor, but their long-term effects can be se- 
vere. A sustained loss of enrollment (student exit) or loss of placements (employer exit) 
translates directly into a loss of state revenue (although there may be a delay of a year or 
more before funding levels decline). Thus exit is a powerful, if blunt, instrument for affecting 
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programs. On the other hand, an occasional exit by an individual student or employer has 
more limited impact. 

Similarly, program staff can exert influence on constituents through actions designed to en- 
courage participation (e.g., informal phone calls and formal award ceremonies) and actions 
taken to persuade or change opinions. Most practitioners are genuinely motivated to encour- 
age participation and serve the needs of the business community. Furthermore, failure to 
achieve enrollment and placement targets carries severe sanctions. As a result, most pro- 
gram staff try to maintain positive relationships with employers and the community and re- 
ward participation as much as possible.^ To a certain extent, program staff also strive to 
control their communities, both so they can address concerns before those concerns become 
larger problems and so they can regulate the demands placed upon the program. 

Reform 

The principal justification for outcome-based accountability is its ability to sustain high- 
quality programs and promote reform in programs that are not producing desired results. 
Reform is the way in which the administration responds to influence from constituents and 
to internal feedback about program success. Incremental changes to course Hoa«s. student 
learning objectives, instructional procedures, the structure of internship programs, etc., fre- 
quently occur in response to suggestions from employers or students or in response to formal 
recommendations from industry advisory committees. Similarly, instructors often make 
small changes to lessons and tests in response to student concerns voiced directly to them. 
These are some of the obvious ways constituents' influence leads to program reform on a 
small scale. 

Occasionally major changes, such as the initiation or elimination of a training program, oc- 
cur. Although less frequent, these changes also are prompted by information and influence, 
e.g., lowered enrollments, increased demand from employers, and changes in the workplace. 

Program reform falls clearly within the scope of administrative control. Change does not oc- 
cur unless program administrators permit it In many cases, responsibility is delegated; of- 
ten it is shared; but ultimately it is up to administrators to take responsibility for reform. 
The accountability system provides information and motivation for program improvements, 
but actual reform depends on the effectiveness of the administrators. 

State and Federal Regulations and Funding 

All the actions described above are influenced in direct and indirect ways by state and federal 
regulations and funding mechanisms. For example, the potential negative consequences of 
student or employer exit derive from eventual state sanctions, as well as from the actions of 
the local constituents themselves. Similarly, state and federal requirements for collecting 
and reporting data on placements give such measures increased importance, and state re- 



^State guidelines frequently require that each program have an advisoiy committee and consult with the 
committee regularly. In many cases, advisory committees must approve program objectives as a condition of state 
funding. However, most, if not all, administrators and program staff genuinely desire feedback that helps to 
improve the program. They may resist change that requires effort, as many people do, but they will overcome their 
resistance if they believe the change improves the program. 
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quirements regarding advisory committees increase the value placed on these bodies by pro- 
gram administrators. In many ways, some obvious and some subtle, the external framework 
for vocational education created through regulations and funding mechanisms colors ac- 
countability relationships at the local level. 

At the federal level, Perkins II emphasizes outcome measures to the almost complete exclu- 
sion of input or process measures. As we noted above, there are drawbacks to th:s approach 
from the perspective of program improvement. Although they may satisfy policy r^iakers* 
needs for monitoring, aggregated outcome measures alone do not satisfy practitioners' needs 
for information to guide program improvement. Furthermore, by deemphasizing input and 
process measures, such requirements may even reduce the local capacity to act effectively. 

State regulations also can create pressures that reduce local accountability. For example, 
attendance-based funding can send the wrong message to program staff, focusing their atten- 
tion on short-term success (filling seats) rather than long-term outcomes (employment of 
graduates). In contrast, program-based funding**^^ provides a more consistent basis for build- 
ing local capacities, while still allowing states to exercise final control of resources tied to 
long-term success. 

It is clear that one of the purposes of Perkins II is to foster program improvement through lo- 
cal responsibility, yet other actions on the part of policymakers sometimes run counter to the 
best interests of local accountability. 

Limitations 

The local accountability system described above is overly simplified in that it disregards po- 
tential differences between the perspectives of program staff and constituents. Different ac- 
tors in the accoimtability system have different points of view. They may disagree on goals 
and objectives, and they certainly will assign different priorities to them. In reality, all 
stakeholders do not act in concert, and they often represent conflicting desires and pressures 
on program administration. 

Each major constituency — students, employers, instructors, and parents — ^has different needs 
and voices concerns about different program outcomes and features. Furthermore, other, 
more diffuse constituencies contribute to the cacophony of voices trying to influence pro- 
grams. For example, the local community supports the school through taxes and can voice its 
approval or disapproval at the ballot box. Yet the community has no organized representa- 
tion and seldom becomes involved in individual programs unless rallied by an issue or prob- 
lem. Similarly, trade associations or professional organizations can be directly affected by 
the school's activities and may wish to promote their own point of view. Resolving such con- 
flicts is a test of administrative skill, and it also may be an activity in which technical assis- 
tance can be useful. 

Similarly, the description of the model should not be interpreted to mean that all programs 
have effective accountability mechanisms. There are many practical constraints in the cur- 



some states, annual funding is awarded in a lump sum, independent of the number of students actually 
served. If enrollment in a particular program remains low year after year, funding for the program will be 
terminated. However, daily, weekly, or monthly fluctuations do not aflect funding. 
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rent setting (Stecher and Hanser, 1992). In some cases, goals are sufficiently broad or vague 
that it is impossible to know whether they have been achieved. Constituents do not always 
understand or support the goals, and seldom are priorities among goals clearly established. 
Measures are not always consonant with or sufficient to encompass goals, and their technical 
quality may not be adequate. Feedback between program staff and constituents may be in- 
frequent or inaccurate. Opportunities for influence are often limited. Organizational reform 
mechanisms sometimes falter due to regulations, lack of resources, concern with short-term 
outcomes, or lack of strategies for change. Overcoming these practical limitations is one of 
the challenges LEAs and SEAs face in trying to implement the broader accountability model 
envisioned by Perkins II. 
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3. EVALUATING AND STRENGTHENING LOCAL 
ACCOUNTABi JTY SYSTEMS 



Perkins II places new demands on SEAs. In the broad context of developing and maintaining 
programs that meet statewide vocational education needs, SEAs are responsible for monitor- 
ing local program improvement efforts and providing technical assistance to local programs. 
We believe that from the state's perspective, the most important questions are. How well are 
local systems operating and what can be done to improve them? That is, are the local ac- 
countability systems working effectively to improve local programs, and if not, what can the 
state do to assist them? 

In this section we explore each of the components of local accountability systems from the 
perspective of state educational administrators assigned the role of monitoring and providing 
technical assistance. We discuss approaches for evaluating the effectiveness of a local ac- 
countability system, and we offer some ideas about how states can act to strengthen local 
functions. Our suggestions are based on interviews with state and local staff and our analy- 
sis of the functioning of local accountability systems. Our interviews revealed many aspects 
of local accountability that can inform the evaluation and technical assistance process. Addi- 
tional ideas to enhance local accountability could be drawn from the literature on school im- 
provement through program evaluation, innovation, and school restructuring (Alkin, 1985; 
Loucks and Hall, 1977; Council of Chief State School Officers, 1989). 

Much may be gained from the simple recognition that local accountability systems exist; by 
giving credibility to local accountability, one may improve it. By acknowledging these func- 
tions, delineating them clearly, and sharing the vocabulary of local accountability with pro- 
gram staff and constituents, SEAs can go a long way toward improving such systems. Pro- 
gram staff and constituents can be buoyed by the knowledge that they, rather than external 
forces, are the masters of their own accountability. 

GOALS 

Educational goals are elusive, not always explicit, and not easily explicated (Stake and 
Denny, 1969). Consequently, it may not be possible to identify all the goals that are operat- 
ing in a particular program context States should not expect to find accountability systems 
delineated explicitly, as they are in the idealized model in Figure 2.1. Nor should they try to 
impose formalized standards for goals and goal-setting, such as annual goal clarification ex- 
ercises or lengthy documentation of high-level goals and linked objectives. In many cases 
these efforts would be counterproductive. They are the sort of overly general mandates th^t 
produce low-level compliance rather than commitment and use. 

Goals should be evaluated in light of their functions. Goals serve principally as organizing 
mechanisms for program planning, evaluation, and reform. They should be judged less on 
the basis of content and format and more on the basis of relevance and use. That is, it is less 
important that the goals be stated in behavioral terms and contain an objective relating to a 
specific domain — say, administrative efficiency — than that they be endorsed by employers 
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and used as the basis for choosing measures. Consequently, states should not adopt an 
overly formalized stance toward reviewing goals. 

Instead, states should look for the functional equivalent of shared, measurable, updated 
goals and objectives. They should seek the answers to questions such as: 

• Does the program have goals, and do administrators and staff agree on what they are? 

• Do constituents have goals for the program, and do these goals match those of staff? 

• Are the goals measurable, i.e., can one determine if they have been achieved? 

• Are the goals reasonable in light of local program experience and the experience of pro- 
grams in other areas or other states? 

• Does the program examine its purpose regularly to be surt it is on target? 

• Can the program determine whether it is making progress? 

• Do stakeholders and local experts have a role in giving direction to the program, i.e., in 
setting goals? 

• Has the program established priorities among its goals? 

• When allocating scarce resources, do administrators use program goals to gizide decisions? 

These questions seems obvious, but understanding goals is not a simple process. As Worthen 
and Sanders (1973) cautioned. 

There is some distinction between what educators propose to pursue, what they see themselves 
pursuing, and what they in fact do pursue. It is not a matter of deception. Evaiuators, educators, 
all human beings, have enormous difficulties in reporting the sum and sweep of their objectives. 
We ail have goals, and we consciously give priority to some goals over others. But we have few re- 
liable ways to report them to others, or even to reveal them to ourselves (p. 332). 

There are many actions states can take to strengthen goals and goal-setting. First, by 
merely asking LEAs questions about the nature and status of program goals, SEAs alert pro- 
grams to the clarity or vagueness of their mission. Second, by sharing effective goal state- 
ments and goal development activities from other institutions, states can provide useful 
models. Third, by stressing functions over formality, and allowing programs to develop their 
own goal-setting mechanisms, SEAs invest participants more deeply in the process and en- 
sure that the results will be more meaningful at the local level. Fourth, by pooling resources 
to develop core knowledge centrally (e.g., statements of the knowledge, skills, and abilities 
required for specific occupations), states can relieve local programs of oppressive burdens and 
yet permit them to tailor descriptions to their local needs. 

MEASURES 

Measures are essential to accountability. Without methods of tracking program status, one 
cannot make judgments about program quality or take appropriate action. Fortunately, it is 
possible to form impressions without quantifying outcomes, so reasonable action can be taken 
in the absence of aggregated, quantitative information. However, impressions can be subjec- 
tive, idiosyncratic, and volatile, and they may provide only a precarious basis for action at a 
programmatic level. Multiple sources of information are to be encouraged, and they should 
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include explicit quantitative data to portray the accomplishment of goals. In addition, it is 
worthwhile to have information that is directly relevant to program planning, including data 
regarding processes and inputs. 

Measures serve specific functions, and their adequacy can be determined by exploring how 
these functions are performed in a particular program. Questions that address the functions 
measures perform include:^ 

• Does the program know whether it is accomplishing its goals? 

• What information is available that is relevant to goal attainment? 

• How accurate and meaningful is this information? 

• Is the information available to those who need to know? 

• Do constituents share the impressions of staff? 

• Do constituents understand the meaning of the information they receive? 

• Are there important goals the program is not measuring and consequently is not consider- 
ing adequately in planning? 

• If the program fails to meet outcome objectives, does it have information to help it make 
improvements? 

There are many ways states can help programs strengthen their measures. First, programs 
may need assistance in choosing measures effectively. External agents can help local pro- 
grams ensure that measures relate to goals, are of high quality, and can be interpreted 
meaningfully. Second, states can provide technical assistance to help programs improve the 
measures they have and the manner in which they report them. Third, states can encourage 
selective sharing of measures and measurement approaches. Once a measure or an assess- 
ment strategy has proven to be effective, SEAs can share this information with other pro- 
grams or can provide regional or statewide forums for contact and dissemination. Fourth, 
states can provide thp impetus for groups of programs to pool their expertise in developing, 
identifying, or implementing measures related to common goals. Finally, states can use 
state-level resources and expertise to develop, implement, or disseminate measures that may 
benefit local programs.^ Statewide development of occupational competency tests is a good 
example of this type of activity. 

FEEDBACK 

To determine whether a program provides effective feedback, states must consult with staff, 
students, ar:d constituents, and must review the documents and reports that are made avail- 
able to various groups. The key questions are: 

• How do constituents and staff know whether or not the program is effective? 



^Measures should be closely linked to goals; therefore, the questions one asks regarding measures are similar to 
those regarding goals. 

^Alternatively, local districts might be asked to pay a small "tax* into a central fund to pvovide such supple- 
mental technical services. 
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• What kind of information about program success do they receive? 

• How often is this information made available? 

• How useful is the information to them? 

• Do constituents and staff have questions about the program that are not answered? 

The key to strengthening feedback is to increase access to, and use of. information about pro- 
gram performance, particularly information about goal attainment derived from objective 
measures. Program administrators control most of these data, so increasing feedback to con- 
stituents requires changing administrators' behavior. In some cases, it may be sufficient to 
educate administrators about alternative strategies for sharing information with con- 
stituents. In other cases, administrators may perceive greater access to information as a 
threat to their position. It is difficult to overcome these perceptions, but administrative staff 
development illustrating the value of collaboration and the use of staff, student, and con- 
stituent input for program improvement may be an effective starting point. Methods for 
planning, gathering, and disseminating information in more ^^utilization focused" ways are 
described in Patton (1978). 

INFLUENCE 

By exerting influence on one another, staff and constituents put "teeth" in^x) the accountabil- 
ity system. Constituents voice their concerns and reactions to information and their impres- 
sions about program performance, or they discontinue their participation and exit the arena. 
Staff ake constituents aware of their own concerns and the constraints under which the 
program operates. To judge the effectiveness of these two types of influence, one needs an- 
swers to questions such as: 

• How satisfied are constituents with the program? 

• How satisfied do program staff think constituents are with the program? 

• Do constituents have avenues for making their opinions known to staff? 

• If constituents are dissatisfied, what do they do? 

• In what ways are constituents involved in program review and planning? 

• Which constituents do program staff consult most oflen? 

• Has the program ever changed because of input from constituents? 

• What positive and negative consequences are associated with constituent exit? 

• How real do program staff perceive these consequences to be? 

• How do program staff make their concerns known to constituents? 

• Do program staff try to manage constituents' reactions? 

• Do constituents feel program staff are responsive to their concerns? 

One way in which influence can be strengthened is by changing the structural features of 
program funding or governance. For example, linking state funding to enrollments, comple- 
tion rates, and/or placement rates is a common method for strengthening the consequences of 
constituent exit. In a more subtle way, some states require programs to have their curricula 



ERIC 



3i 



20 



approved by industry advisory committees annually in order to receive state funding. This 
institutionalizes one avenue through which constituents can have a direct effect on program 
design. States can also adopt rules and funding mechanisms that grant greater power to 
constituents. 

There are other ways states can intervene to increase the level of collaboration between pro- 
grams and constituents. Cooperation may be absent because neither administrators nor con- 
stituents have a framework for collaborative change that they can use to structure their in- 
teraction; both parties may be skeptical about the motives and actions of the other. In some 
instances, the parties have assumed adversarial roles in the past that are difficult to over- 
come. Education is the key to changing this situation and increasing the responsiveness of 
each party to the concerns of the other. Staff development, particularly staff development 
that shares models of successful cooperation at the local level, is a good starting place. 

PROGRAM PLANNING AND REFORM 

Our model of an accountability system includes the capacity for program reform as an essen- 
tial element. Although some would conceive of reform as a response to accountability, we in- 
clude it in the model because Perkins II so clearly justifies accountability in terms of program 
improvement. Without a mechanism for change, the finest goals, the best information sys- 
tem, and the most influential constituents will have little impact. Forces can be aligned to 
push for change, but actual improvement will depend on the willingness and support of the 
administration. The kinds of questions that might be asked to judge a program's capacity for 
reform include the following: 

• How are new programs initiated? 

• Under what circumstances are progprams discontinued? 

• What have administrators done in the past to improve programs? 

• What would program administrators do if they learned a program was having problems 
(e.g., enrollment was low, a high proportion of students were not completing the program, 
the placement rate for the program was low, staff morale was low)? 

• Do individual programs conduct internal reviews, and if so, how are they conducted? 

• What role do instructors and department administrators play in curriculum review and re- 
form? 

• How do programs use input from employers and industry representatives? 

• If students are dissatisfied with a class or a program, what happens? 

Traditionally, program planning and reform activities fall within the domain of administra- 
tive action. But administrators may have a Hmited repertoire of options, which reduces their 
ability to respond to deficiencies that arise. As noted earlier, an overemphasis on output can 
contribute to this difficulty, because output data provide few clues for program improvement. 
It is important to temper enthusiasm for outcome measures with the recognition that under- 
standing processes and inputs is necessary for institutional reform. 

There are several ways states can promote more effective program planning and reform. 
SEAs can identify effective administrators and share their skills with those in need of as sis- 
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tance. Superintendents, principals, and/or department heads who have developed useful 
strategies for program improvemer.t or successfully confronted difficult program challenrjes 
are among the state's most valuable resources. It is possible to encourage collaboration 
between these individuals and others in need of such skills. Second, SEAs can establish net- 
works to promote educational change by building communities of teachers and administra- 
tors with common concerns (Lieberman and McLaughlin, 1992). Such networks can be fos- 
tered through meetings, newsletters, or electronic bulletin boards supported by the state. 
Third, states can engage in more trarlitional staff development activities focused on the role 
of performance measures, other local data, and constituent input in program improvement. 

SUMMARY 

In considering their monitoring and technical assistance role, state offices of vocational edu- 
cation need to be aware of the structure and functions of local accountability systems, which 
offer great potential for improvement of vocational programs. States can take an active role 
in improving local accountability systems by reviewing and offering guidance to strengthen 
goals and measures, to increase feedback and influence, and to improve program planning. 
There are many instances in which it might be advantageous to promote greater sharing and 
consortium-building among local programs. State vocational educators also need to be cog- 
nizant of the constraints imposed by state regulations and the influence exerted by state 
policies. Their goal should be to align all these pressures in support of local accountability. 
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4. CONCLUSIONS AND RECOMMENDATIONS 



This study was undertaken prior to the deadline for states to implement the standards and 
measures provisions of Perkins II. At that time, there was widespread concern about these 
provisions and about the impact of outcome-based performance standards on SEAs and 
LEAs. This study examined the use of measures and data systems at the local level and ex- 
plored the possible effects of Perkins II on local accountability. Because of the nature of the 
problem we chose to address and the methods we used, our results are qualitative rather 
than quantitative. We did not set out to conduct a census of how SEAs and LEAs were react- 
ing to Perkins II, ^ but rather to understand how local information and decisionmaking 
systems (specifically, accountability systems) operate, and to gain a sense of what effects 
Perkins II might have on them. It was toward that end that we structured our field inter- 
views and analyzed the legislative history and regulations governing Perkins IL 

In an earlier publication (Stecher and Hanser, 1992), we described a model of local account- 
ability in detail, examined the way it would operate under ideal circumstances, and discussed 
the limitations we observed in local accountability systems in practice in various sites across 
the country. The focus of this report is quite different; we have not attempted to duplicate or 
repeat our earlier work, but rather have extended it. Thus, we first summarized the provi- 
sions of Perkins II that define the roles of states and local programs vis-^i-vis local account- 
ability and program improvement. Next we revised our earlier model and considered the 
ways in which local accountability systems could function as program improvement tools to 
fulfill the demands of Perkins II. Finally we reviewed the role of states vis-a-vis Perkins II 
accountability and suggested approaches states might use to foster local accountability sys- 
tems. 

In our judgment, local accountability systems are the linchpin of Perkins II — local program 
improvement hinges upon their success or failure.^ Furthermore, the states are clearly 
charged with being what we might describe as meta-evaluators — responsible for evaluating 
how well local systems perform rather than for how well local programs perform. Because 
these responsibilities are new, for both the states and the local systems, we have presented 
our analysis of how states can monitor and strengthen system performance. 

Finally, we have come to several conclusions with regard to the effects of Perkins II and have 
developed a set of recommendations that we believe would strengthen the larger vocational 
education system that Congress and the states have been building. 

CONCLUSIONS 

We draw five conclusions based on our analysis of federal policies, local accountability sys- 
tems, and the interactions between them. The first three conclusions highlight the existence, 
use, and improvement of local accountability systems compatible with Perkins II: 



^See for example, McDonnell and Zellman (1992). 

^Although tremendous resources have been devoted to developing measures and standards, and those elements 
may therefore be considered to have paramount importance, they are but components of an accountability system, 
the whole of which must function effectively for program improvement to occur. 
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1. Most vocational programs have functioning local accountability systems consistent with 
the model envisioned by Perkins IL 

2. Local accountability systems can be effective tools for program improvement if practical 
obstacles can be overcome. 

3. States can do much to evaluate and strengthen local accountability syr^ems. 

The final two conclusions focus on the direct effects of Perkins II on local vocational educa- 
tion systems: 

4. Neither SEAs nor LEAs are fully prepared to carry out their responsibilities vis-6-vis 
accountability under Perkins II. 

5. Perkins II may overemphasize outcome-based standards and measures, lessening the use 
of input- and process-based measures that are valuable for program improvement. 

Functioning Local Accountability Systems Exist 

Both the nature of vocational education itself and the manner in which it traditionally has 
been implemented create fertile conditions for local accountability (Stecher and Hanser, 
1992).^ There is ample evidence of functioning accountability systems in local vocational 
programs. Those programs have goals and objectives (usually in terms of enrollment, com- 
pletion, and job placement), require the direct involvement of constituents in program review 
and design (usually employers and/or trade associations), and are subject to rewards and/or 
sanctions based upon performance (usually in the form of program growth or discontinua- 
tion). As a result, constituents maintain influence over programs and work to promote the 
needs of the local community. 

Although the existing local accountability relationships we observed are not as elaborate or 
refined as the abstract model presented in Figure 2.1, they perform the key functions of an 
accountability system and are generally consistent with the philosophy embodied in Perkins 
II. The law emphasizes the use of data for program improvement and places primary re- 
sponsibility for reform at the local level, and at each of the sites we visited, decisions to 
launch, discontinue, or modify programs were initially reached at the local level before any 
state intervention was contemplated."* State guidelines and the threat of intervention may 
have created an atmosphere that fostered quick local action. To the best of our knowledge, 
local decisions are made on the basis of enrollments, placements, or a demonstrated local 
training need. Thus, the major elements of accountability are already operative at the local 
level, consistent with Perkins IL 

Local Accountability Systems Can Aid Program Improvement 

Evidence from the sites we visited suggests that many of the Perkins II accountability provi- 
sions can promote local program improvement. The model presented in Figure 2.1 and the 



Stecher and Hanser (1992) describe local accountability systems in generic terms and discuss some of the 
practical limitations that reduce the effectiveness of these systems. 

"^MoBt states maintain formal approval or veto authority, but decisions are effectively made at the local level. 
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discussion in Section 2 illustrate how this might be accomplished. Goals that are explicit and 
widely shared serve as focal points for program review and reform. Measures that are ade- 
quate to reflect goal attainment provide useful information about program status. Feedback 
that is timely and clear permits staflF and constituents to judge program success. Adminis- 
trators who welcome constituents* expertise and respect their opinions build a solid founda- 
tion for collaborative program reform. Skilled leaders can reduce other obstacles to program 
improvement, such as tensions between constituent groups with conflicting goals and needs 
and contradictory pressures from federal and state regulations. 

Research suggests that the most potent factors leading to local program improvement are in- 
creased local autonomy and increased stakeholder involvement in all aspects of program de- 
sign and reform (David, 1987; Leviton and Hughes, 1981; van den Berg, 1992). Researchers 
studying effective secondary schools have found that good site-based management reflects lo- 
cal independence rather than compliance with procedural requirements (Hill and Bonan, 
1991). Similarly, studies of school innovation emphasize the need for local adaptation of re- 
forms rather than strict adherence to models (Berman and McLaughlin, 1978). Effective 
policies are those that foster schools as strategic organizations, managing and guiding them- 
selves, hot merely as "executing organizations" carrying out the policies of others (van den 
Berg, 1992). The "local" character of the accountabihty systems we observed permits a high 
level of direct staff and constituent involvement, conditions amenable to effective program 
improvement. The Perkins II model of accountability is consistent with this research in its 
emphasis on local responsibility for program improvement. Overall, it would seem that a lo- 
cal accountability system has the potential to be an effective instrument of educational im- 
provement. 

SEAs Can Evaluate and Strengthen Local Accountability Systems 

Perkins II limits states' roles in the accountability process. SEAs' primary responsibility is 
to establish a system of standards and measures and provide technical assistance to LEAs, 
who perform the program improvement function. Only in the event of continued poor results 
on the part of LEAs do states become directly involved in program planning. 

Nevertheless, it is clear that states can use their oversight and technical assistance roles to 
good advantage by focusing on the entire local accountability system rather than merely on 
local outcomes. SEAs that understand how local accountability systems function can play a 
role in accountability system improvement, which should lead, in turn, to better local pro- 
grams. This enlarged state role involves both evaluation and action. SEAs can conduct criti- 
cal reviews and offer suggestions for improvement of all components of the accountability 
system. Effective questioning can go a long way toward improving local accountability sys- 
tems. Once good questions have been asked, local programs may engage themselves in solu- 
tions; certainly that is in keeping with the spirit of Perkins II. In addition, SEAs can share 
information about effective local practices observed within the state and in other states, can 
promote consortium-building among local education agencies, and can align state policies 
with federal regulation in support of local accountability. 



3u 



25 



Neither SElAs nor LEAs Are Fully Prepared for Their New Roles 

Perkins II assigns new roles to SEAs and LEAs which create challenges for both types of or- 
ganization. The initial challenge SEAs must face is a shift in the balance between their 
traditional functions of oversight and improvement (OERI State Accountability Study Group, 
1988). In the past, SEAs were directly responsible for program improvement, and they were 
given resources to use flexibly to support LEAs. Under Perkins II, their responsibilities have 
shifted toward oversight, and their resources have been reduced.^ Whether SEAs now have 
the wherewithal to provide the necessary oversight remains to be seen. 

Moreover, the nature of the oversight role has changed. SEAs have always monitored pro- 
gram performance to some extent, but they have little or no experience in overseeing local ac- 
countability systems. Few SEAs have the knowledge to evaluate local accountability systems 
or the experience to help LEAs strengthen those systems. This limitation is important be- 
cause these are the systems through which program improvements will come about. SEAs 
have much to learn about local program improvement if they are to fulfill their support func- 
tion in the accountability context created by Perkins II. Furthermore, the combination of re- 
duced budgets and changing requirements makes it increasingly difficult for SEAs to respond 
to local needs.^ 

LEAs face equally great challenges in fulfilling their roles under Perkins II. Local policy- 
makers must now assume the primary responsibility for achieving outcome standards, diag- 
nosing program deficiencies, developing remedies, and putting them into action. Although 
LEAs were responsible for programmatic functions in the past, they were not held to strict 
standards. Many states had enrollment requirements, a few had fixed standards for place- 
ment, and almost none had standards for student achievement, skill acquisition, and the 
other kinds of measures required by Perkins II. Consequently, local programs have little or 
no experience in being directly accountable for a broad spectrum of outcomes. Initially, they 
will need assistance in aligning their programs and data systems with the statewide mea- 
sures; later they will need assistance in using the information these measures provide as 
guides to program improvement. 

Local programs also have limited experience in promoting the active involvement of con- 
stituents that is crucial to the type of accountability system described here. Most programs 
have industry advisory committees that play some role in program development, but few in- 
vite the broad participation of constituents or cede to constituents the degree of power and 
involvement required by the local accountability model proposed here. To make local ac- 
countability work, LEAs may need to develop new capabilities and may need a great deal of 
advice. It is not clear where they can find useful insights into these processes, for SEAs may 
be as naive as they are. 



°*It is the Committee's clear intent that the primary use of the federal dollar should be that of direct services. In 
the Committee's view the five percent limitation on administration with the required match achieves that objective" 
{U.S, Code Congressional and AdministrcUive News, 1991, p. 1197). 

^Reported by state directors at the 1992 annual meeting of the National Association of State Directors of 
Vocational and 'Technical Education Consortium. 
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Perkins n May Overemphasize Externally Developed Outcome-Based Standards 
and Measures 

Although Perkins II does much to strengthen the role of local programs as agents of reform, 
it also calls for some approaches that may reduce the effectiveness of local programs and the 
systems that support them. Some of the provisions that are designed to increase attention to 
program performance and to promote greater local responsibility for program improvement 
may be at odds with current understanding of effective school reform. For example, out- 
comes may be a reasonable basis for monitoring program performance, but they are a poor 
basis for guiding program improvement. Information about the causes of failure to meet 
goals is far more useful for school reform than information about effects (King and Pechman, 
1984). It is important to have information about features that affect performance (e.g., in- 
structional content and quality) as a basis for developing program improvement strategies. 

Perkins II's requirements for data collection and its strong emphasis on outcome measures 
are likely to drive out supplemental measures like those mentioned above and may narrow 
the scope of information available to practitioners and constituents. This conclusion is based 
on evidence that outcome-based accountability has had deleterious effects on education. In 
particular, test-based accountability led to narrowed curriculum, reduced time devoted to 
nonmeasured content, and inappropriate preparation (Koretz et al., 1991; Rottenberg and 
Smith, 1990; Cannell, 1987), Similarly, outcome-based accoimtability in the Job Training 
Partnership Act (JTPA) led to changes in selection procedures and narrowing of services 
(King, 1988). It is not too great a leap to suggest that an outcome-based system of standards 
and measures will create similar pressures on vocational education, i.e., will cause a reduc- 
tion in attention to and resources for information about nonoutcome elements of vocational 
programs. This is precisely the information that will be most important if reforms are 
needed. 

More generally, the vision of program improvement through outcome-based accounta- 
bility presumes that both individuals and organizations act in a ratii^nal manner, given the 
incentives and pressures in their environment. Extensive research on school improvement 
and sc' ool reform suggests that neither presumption is valid (David, 1987; Fullan, 1982; 
Huberman and Miles, 1984; van den Berg, 1992; Weich, 1976; Weiss, 1980). For example, a 
rationally behaving school administration would use relevant data to improve program 
design and decisionmaking. Yet educational organizations do not appear to behave in this 
manner. Data are seldom a catalyst for change (David, 1987), and mandated data-collection 
in particular has had limited impact on program reform (Alkin et al., 1979; David, 1981). 
When mandates prompt data-collection, schools tend to use data only to "signaP their 
compliance, not as the basis for informed program improvement (Zucker, 1981). And voca- 
tional educators, like their counterparts in general education, have limited experience with 
the use of data to manage or improve programs (Hoachlander et al., 1992). 

The strong emphasis in Perkins II on uniform outcome-based standards and measures within 
a state*^ can also pose a problem because states are not uniform from a programmatic point of 



is important to note that Perkins II permits LEAs to make exceptions to statewide standards based on local 
conditions, provided those exceptions conform to the assessment criteria in the state plan. However, this provision 
is not widely discussed. At the 1992 annual meeting of the National Association of Sute Directors of Vocational and 
Technical Education Consortium, one of the authors asked a group of state directors whether local programs knew of 
this provision of the act. One answered, "ShhW," and a second replied, Tm sorry I didn't hear the question." The 
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view: Community and population characteristics can vary from locale to locale, socioeco- 
nomic dififerences between neighborhoods can be great, and state labor markets can be ex- 
tremely heterogeneous. Under these circumstances, local schools need to be able to adapt the 
goals they set for themselves, the programs they offer, the manner in which they provide in- 
struction, and the nature of their placement assistance and follow-up services. Although 
uniform standards and measures do not constrain schools to offer the same programs in the 
same ways, they may discourage diversity. 

In fact, the preference for uniform standards and measures sends the wrong signals regard- 
ing diversity on a number of different levels. The preferc^nce for uniform measures will drive 
the adoption of uniform goals. Given limited LEA reSs'^urces for program design, data- 
collection, and reporting, the Perkins II requirements for common measures are hkely to 
diminish the stature of unique local goals. At a more subtle level, systems of common 
standards suggest that programs should be similar rather thav> different. The goals and 
measures that will now receive reduced emphasis are precisely the ones that give programs 
their unique local flavor.® These are a few of the potentially negative consequences of 
statewide standards and measures that need to be monitored carefully as states implement 
the requirements of Perkins II. 

A related concern is that Perkins II is somewhat contradictory with respect to local respon- 
sibility and efficacy. Except in special circumstances, local programs are being required to 
adopt external definitions, standards, and directions at the same time that they are being re- 
quired to take on greater responsibility. Perkins II "empowers" LEAs with responsibility for 
program design and improvement, but it usurps some of their powers. It is not clear how 
SEAs and LEAs will respond to these conflicting signals. Perhaps SEAs and LEAs are so 
conditioned to respecting the letter of tne federal law that they will find nothing troubling in 
adapting to these small counterpressures, but the issue is worthy of some continuing atten- 
tion. 



RECOMMENDATIONS 

Implementation of Perkins II should result in many positive consequences for vocational ed- 
ucation at the state and local levels. However, ample room for improvement exists. We rec- 
ommend three modifications to the federal policy embodied in Perkins II and its associated 
regulations: 

1. Encourage SEAs to monitor and strengthen local accountability systems, not merely 
program outcomes. 

2. Increase the emphasis on, and support for, accountability-related capacity-building at 
state and local levels. 

3. Broaden the emphasis in Perkins II on outcome measures to include measures useful for 
program improvement. 



others nodded in agreement. Until the new standards and measures are implemented and one or two annual review 
cycles are completed, we will not know how often such exceptions are taken or what reactions SEAs will have. 

®And, as was discussed above, they are the goals and measures that provide the kinds of information needed for 
program miprovement. 
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SEAs Should Monitor and Strengthen Local Accountability Systems 

Current federal vocational education legislation emphasizes outcome standards and assigns 
SEAs the responsibility for monitoring program performance according to these standards. 
One goal of Perkins II is to use local accountability as a program improvement tool. If this 
model of vocational education is to be effective, local programs will have to become self- 
improving organizations. This is an essential feature of the Perkins II strategy. 

Unfortunately, SEAs have little experience in evaluating accountability systems or fostering 
a program's capacity for self-improvement. The regulations direct SEAs to intervene if LEAs 
fail to make progress toward meeting standards and to help the LEAs develop plans for im- 
provement. To do this, SEAs need to be able to reflect critically on local actions, including 
program goals, the links between curriculum and workplace needs, instructional practices, 
and student and employer satisfaction. SEAs must evaluate the local accountability systems 
if they are to develop plans for program improvement By monitoring the components of local 
accountability systems on a regular basis, SEAs will gain insights into strategies they can 
use when programs fail to achieve their goals. This broader conception of the state's over- 
sight role should be incorporated into federal policies, possibly by delineating specific 
accountability-related support activities in which states should engage. 

Emphasize Capacity-Building 

As noted above, Perkins II creates new roles for SEAs and LEAs. The changes are not triv- 
ial, and both groups need preparation to carry out the new functions assigned to them. For 
example, states could not implement the Perkins II requirements for statewide standards 
and measures without considerable assistance. The NCRVE has developed training materi- 
als (Hoachlander et al., 1992) and conducted workshops to help states implement systems of 
standards and measures.^ 

Similar capacity-building activities are needed to help states meet the needs of LEAs and de- 
velop expertise in promoting local accountability. In the short term, SEAs are confronting 
increasing demands for technical assistance with decreasing levels of discretionary funds. 
Some are evolving strategies that appear to be effective in many situations (e.g., establishing 
local school consortia to pool resources or share expertise); others are struggling to cope. A 
few states have already embarked on long-term educational reform programs that embody 
many of the decentralized features of Perkins II. For example, Georgia, Indiana, and Penn- 
sylvania have developed management systems and evaluation models that establish links be- 
tween outcomes and local services (Curtis, 1991; Hull, 1987; Jordan and Lyons, 1990). And 
Alaska has implemented a program review process that relies on local vocational advisory 
councils (Kirts, 1988). However, most states have not undertaken such reforms, and their 
SEAs will need assistance as the Perkins II accountability cycles occur. 

At the LEA level, Perkins !I multiplies the number of institutions responsible for program 
review and evaluation. This in turn multiplies the number of people who need expertise in 
asking questions; selecting, collecting, <Knd analyzing data; interpreting data; and using in- 



^orkahopg on performance measures and standards for vocational education were held in March 1992. 
Handouts of overhead slides and other materials used in the workshops as well as videotapes of the sessions are 
available from the NCRVE, M.iterials Distribution Service, Western Illinois University. 



40 



29 



formation for program decisions. These requirements will create a secondary level of demand 
on states that many will be unable to fill. The federal response should take the form of in- 
creased capacity-building at the state level. Specifically, additional training should be pro- 
vided to increase states' capability to support the use of measures and standards for program 
improvement and the enhancement of lOcal accountability systems. 

Broaden Measurec to Include Those Relevant for Program Improvement 

Some of the conditions of Perkins II restrict the scope of action of local programs in ways that 
may reduce their capacity for program improvement. One change that might foster better 
programs would be to broaden the scope of the measures that are part of the state system. A 
related change would be to encourage local variation in selecting measures to report to 
states. 

The change that is likely to offer the greatest payoff in terms of program improvement is a 
reduction of the solitary emphasis on measures of program outcomes. Admittedly, the em- 
phasis on outcome-based accountability is one of the most important provisions of Perkins II. 
It redirects attention to the real purposes of the legislation, which may lead to better out- 
comes for students. However, in making this change, the authors of the legislation over- 
looked one of the most important functions of measures — ^their use as guides for program im- 
provement. To be useful for program improvement, a data system must include information 
on features of the program that are subject to policy intervention (Kirst, 1990). Outcome 
measures do not provide this information. 

The legislation should be changed to acknowledge the value of information about program 
inputs and processes and to encourage the incorporation of such information into the ac- 
countability process. Measures of the content and quality of instruction are a critical basis 
for effective program reform. These measures include variables such as teacher qualifit - 
tions, student and employer satisfaction, pupil-to-teacher ratios, and student course-taking 
patterns. Outcome measures should be required, but unless there is explicit recognition of 
the value of other types of data, alternative measures will be squeezed out of most systems. 

Finally, we recommend permitting local programs greater latitude in selecting alternative 
measures to report to states. When programs choose measures that are meaningful at the 
local level, they increase the chance that the information can be translated into program 
improvement. This choice also increases the involvement of local program staff and con- 
stituents in the evaluation and review process, which may have a positive effect on reform 
efforts. 



ERIC 



41 



Appendix 

THE CHANGING RESPONSIBILITIES OF LOCAL AND STATE 
EDUCATION AGENCIES UNDER PERKINS II 

The 1990 amendments to the Carl D. Perkins Vocational Education Act (Perkins II) place 
many new and increasingly detailed demands upon SEAs and LEAs. Thorough assessments 
of those changes are available elsewhere,^ Here, we focus or the legislative intent underly- 
ing Perkins II and those portions of the Act that relate directly to the roles of SEAs and LEAs 
as agents of accountability for vocational education. 

LEGISLATIVE INTENT OP PERKINS H 

According to the legislative history of Perkins II, the 1990 amendments were designed to 
achieve five goals: 'T^argeting the money to those who need federal help the most, stressing a 
linkage between secondary and postsecondary education, integrating academic and voca- 
tional education, making sure that vocational education programs are state-of-the-art, and 
an adherence to the concept that vocational education is not simply job training, but voca- 
tional education and training. ''^ 

Congress recognized that it could leverage the proportionately small amounts of federal 
money offered under Perkins II to achieve these policy goals; although federal funding ac- 
counts for only a small portion of total spending on vocational education, Congress noted that 
''it is without question the engine that pulls the train."^ Local programs also acknowledge 
the importance of federal vocational education funds. In our visits to schools, we found that 
Perkins money paid for services that schools would otherwise go without, such as learning 
resource centers. 

Research has shown that the intent of previous vocational education legislation was 
"frittered away," with the average grant being only $5,000.* Thus, one of the primary inten- 
tions of Perkins II was to ensure that the bulk of funds will be driven down to the local level 
in amounts large enough to be significant. Only a small percentage is available to the states, 
and that amount is allocated only to provide those services that Congress deems appropri- 
ate.^ 

Congress fully intends to use Perkins II to affect %e role of SEAs in vocational education: 
"The provisions with respect to the responsibilities of the State represent another dramatic 
departure from current law— one the Committee believes will strengthen and expand the role 
the State now plays in vocational education. Perkins II "offers the State a new role and 



^American Vocational Association, Inc. (1990); Hoachlander et al. (1992). 
^U.S. Code Congressional and Administrative News (1991), pp. 1182rr. 
^Ibid., p. 1187. 

^National Assessment of Vocational Education, July 1989. The Carl D. Perkins Vocational Education Act, Sees 
231 to 234, specifies minimum grant amounts that will be made to LEAs. 

^The activities to be performed by the states are more clearly spelled out in 34 CFR Part 400 et al. (1992). 
^U.S, Code Congressional and Administrative News (1991), p. 1193. 
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new powers in monitoring and evaluating what goes on at the local level and in providing 
leadership in tying programs to the State's economic expansion. ... In addition, . . . the State 
plays a critical role in curriculum development, in teacher, administrator, and counselor 
training, and in the training of community-based organizational personnel."^ 

But it is not just the state's role that Congress sought to modify; Perkins II also was intended 
to affect the role of LEAs. "For the first time, [it] drives funds directly to the local educa- 
tional agency, the area vocational school, or the intermediate educational agency charged 
with vocational education delivery. . . . [It] places a strong emphasis on programs which inte- 
grate academic and vocational training, and on program improvement as ongoing efforts to 
deliver state-of-the-art vocational education and training to all students."® 

Congress intends to ensure that, at the local level, schools serve the "economically disadvan- 
taged, individuals with disabilities, limited-English-proficient students and [also use the 
federal funds] for programs to achieve sex equity.'*® The means by which Congress hopes to 
accomplish these goals are especially important because they have an impact on the entire 
vocational education system. 

Perkins II embodies a "three-pronged assault" to achieve Congressional goals: "(1) driving 
money directly to the local level; (2) requiring that each recipient of dollars provide the sup- 
plementary services necessary to ensure full and equitable participation in programs; and (3) 
adding new requirements for monitoring of services at the local level and at the State 
level."^® The details of how Perkins II implements this "three-pronged assault" and the sub- 
sequent effects on local accountability systems are described below. 

PERKINS HAND THE ROLE OF SEAS 

Perkins II places new or increased accountability-related requirements on the states in three 
major areas: (1) developing state plans, (2) evaluating local programs, and (3) providing 
technical assistance to LEAs.^^ 

Developing a State Plan 

Perkins II requires that states develop a comprehensive plan covering an initial period of 
three years and subsequent periods of two years. The state plan represents the core of each 
state's responsibilities under Perkins II and is based on an assessment conducted by the 
state board. The basic role of the states in this regard has not changed from Perkins to 
Perkins II, except insofar as the details of the needs assessment and the requirements for 



^Ibid., p. 1194. 
®Ibid., p. 1190. 

^Although this had been the intent of previous legislation, Congress noted that the letter but not the spirit of the 
law had been followed. 

^^Ibid., p. 1201. 

^^These requirements are contained primarily in Sections 113 to 118 (state plan, state plan approval, state and 
local standards and measures, state assessment and evaluation, program evaluation and improvement, and criteria 
for services and activities for individuals who are members of special populations)- 
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intrastate coordination have been broadened and more fully expounded. These plans are rel- 
evant to accountability because they represent the states* initial attempts to specify skills to 
be transmitted that may later form the basis for assessment and standards of performance. 

Previously, Perkins required states to assess current and projected occupational needs and 
demands of employers within the state and the needs of students . .nd adults (including the 
special needs of the handicapped, disadvantaged, single parents, criminal offenders, etc.) to 
achieve the skills essential for meeting state occupational and skill requirements.^^ states 
were also required to assess the quality of vocational education (i.e., pertinence, responsive- 
ness, technical quality) and to determine the capacity of LEAs to deliver services to meet 
identified needs. 

Perkins II specifies the required contents of the states' needs assessment in greater detail. 
The assessment must now include an analysis of vocational education programs* capabilities 
to provide students, "to the extent practicable,"^^ with: 

(i) strong experience in and understanding of all aspects of the industry the students are prepar- 
ing to enter (including planning, management, finances, technical and production skills, underly- 
ing principles of technology, labor and community issues, and health, safety, and environmental 
issues) [emphasis added]; and 

(^i) strong development and use of problem-solving skills and basic and advanced academic skills 
(including skills in the areas of mathematics, reading, writing, science, and social studies) 
[emphasis added] in a technological setting. 

States are required to formulate a plan that includes a description of the procedures and re- 
sults of the needs assessment and of the progress made in achieving the goals specified in 
previous plans; provides numerous assurances that the state will comply with the provisions 
of Perkins II; and describes the estimated distribution of funds within the state. 

Although the state board bears responsibility for developing the plan, it is to do so in consul- 
tation with the state council^® after conducting public hearings at which interested parties 
can present their views and recommendations. 

The state board is also responsible for including in the plan a description of the funds that 
will be reserved for adult education, postsecondary education, tech-prep education, and sec- 
ondary education. The state board is to consult with representative state agencies in devel- 
oping this portion of the plan, but it bears final responsibility. State agencies may file writ- 
ten objections that the board must consider and may choose to address; the Secretary of Edu- 
cation will take these objections and responses into account in reviewing the state plan. 



^^Cari D. Perkins Vocational Education Act, Sec. 113(aX3). 

^^The phrase 'to the extent practicable" was not in the original House version (BH. 7) but was inserted by the 
Senate during conference on the bill. 

Perkins II, Sec. 113(a)(3KBXi) and (ii). The regulations describing how this assessment is to be carried out are 
given in §403.203 of 34 CFR Part 400 et al. 

^^hese requirements are fully specified in Sec. 113 of Perkins 11 and §403.30-34 of 34 CFR Part 400 et al. 

^^he membership and responsibilities of the State Council on Vocational Education are specified In Sec. 112. 
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Thus, an important role for the states is that of developing objective criteria for assessing 
statewide vocational education and training needs and plans for addressing them. In doing 
so, the states are required to consult widely with various constituent groups. 

Evaluating Vocational Programs 

Although the primary responsibility for evaluating local programs rests with the LEAs,^'' the 
states are given substantial responsibility for evaluating the efficacy of vocational programs 
vis-a-vis statewide needs, as well as for monitoring local program performance. This respon- 
sibility includes needs assessment, a system of performance standards and measures, and a 
model for joint program improvement. 

As noted above, the first responsibility of the state in evaluating vocational programs derives 
from its role in conducting a statewide needs assessment. Not only does this assessment fo- 
cus on the needs of the state for vocational education and training, it also includes an as- 
sessment of the ability of the state's system of vocational education and training to address 
those needs. Prior to conducting this assessment, the state board must develop, in consul- 
tation with a number of representative organizations, "measurable objective criteria" on 
which to base its assessment.^® 

The second responsibility of the state, and perhaps one of its most difficult and important, is 
to "develop and implement a statewide system of core standards and measures of perfor- 
mance for secondary and postsecondary vocational education programs."2° These standards 
and measures are extremely important because they represent the benchmark against which 
LEAs will measure themselves.^^ 

The third responsibility of the state is to enter into a joint improvement plan with those re- 
cipients who for two consecutive years fail to make substantial progress in meeting or exceed- 
ing the standards described above. After one year of failing to make substantial progress, 
the local recipient must develop and implement its own local improveir^ent plan. A year 
later, if the local recipient still fails to make substantial progress, the state enters into a joint 
plan with the LEA. 

Although these requirements are quite explicit, they allow greater flexibility than is at first 
apparent. Perkins II includes a provision that "eligible recipients may make local modifica- 
tions to such systems based on economic, geographic, or demographic factors, or the charac- 
teristics of the population to be served."22 Also, the legislation does not specify that the stan- 
dards must be met, only that recipients must make "substantial progress in meeting the 
standards and measures.*^^ And finally, the responsibility forjudging whether modifications 



I'^PerkinsH, Sec. 117(a). 
l^Ibid., Sec. 116(aX6), (8), and (9). 
^^Ibid., Sec. 116(a). 
20lbi(L, Sec. 116(a). 

^^The development of this system of core standards and measures has received substantial emphasis in a series 
of workshops and publications developed by the NC^RVE. For example, see NCIIVE, 1992; Hoachlander et al. (1992). 

22perkin8n, Sec. 115(a). 

23lbid., Sec. 117(b). 
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are appropriate and whether substantial progress is being made appears to be left to the 
LEAs, in the absence of additional state legislation or regulation. 

These provisions represent the culmination of a lengthy evolution in federal policy — from 
process-oriented to outcome-oriented program management. The emphasis on accountability 
itself is not entirely new in vocational education. As early as 1963, states were required to 
develop state vocational education plans. In 1968, the federal vocational education law 
stated that **due consideration will be given to the results of periodic evaluations of state and 
local vocational education programs, services, and activities. The 1976 amendments to the 
vocational education law required the submission of a formal accountability report, and 
Perkins required •'assurances that the state will develop measures for the effectiveness of 
programs assisted under this chapter in meeting the needs identified in the state plan.^^s 
Although the federal government has been increasingly directive in mandating assessments 
and specifying the contents of those assessments,^*' Perkins II does not extend so far as to 
specify either sanctions or rewards associated with achieving or failing to achieve standards 
of performance. In this sense, Perkins II provides a framework for self-improvement that 
stands in contrast to strict accountability.^'^' 

Providing Technical Assistance to LEAs 

States regularly provide a wide variety of technical assistance to LEAs, but the shifting re- 
sponsibilities of SEAs and LEAs under Perkins II make state assistance to LEAs even more 
critical. Scattered throughout the act are references to the technical assistance the states 
must provide to LEAs,^^ but perhaps the most explicit concerns the state's leadership role. 
Perkins II specifies three distinct activities that states are to engage in: 

professional development activities for vocational teachers and academic teachers working with 
vocational education students, . . . including inservice and preservice training . . . with . . . empha- 
sis on .. . minority teachers; 

development, dissemination, and field testing of curricula; 

assessment of programs . . . including the development of performance standards and measures 
. . . and program improvement and accountability."^^ 

Of these three activities, developing performance standards and measures represents the 
most dramatic addition to SEA technical assistance activities. In addition to developing per- 



^*Vocational Education Amendments of 1968, Sec. 123(aX6XA). 
25perkinB, Sec. 113(bX9XA). 

2^See Hoachlander et al. (1992) for additional discussion of this shift in federal focus. 

2'^David (1987) distinguishes between the use of performance indicators for accountability in a system of 
externally applied pressures for change and their use in a self-improvement process. David claims that for 
indicators to contribute to local improvement, they must be locally defined. 

E.g.. Sec. llKbXlXG) requires the states to provide "technical assistance and advice to local educational 
agencies, postsecondary institutions, and other interested parties in the state, in expanding vocational opportunities 
for women." Sec. 118(d)(2) requires states to provide technical assistance to ensure that members of special 
populations (defined in 34 CFR PaH 400 § 400.4) are "able to directly participate in State and local decisions that 
influence the character of programs under this Act affecting their interests." 
2^Perkins H, Sec. 201(b). 
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formance standards and measures, states are also required to help LEAs implement them. 
States must evaluate their vocational education and training programs and enter into joint 
planning activities with local recipients that fail to make substantial progress in meeting 
standards for two consecutive years. In these circumsta'^ces, Perkins II also requires the 
state and local joint plan to descriL-. "the technical assistance and program activities the 
state will provide to enhance the performance of the eligible recipient."^° The intervention of 
the state at this point is not meant to be "punitive in nature, but rather to encourage an infu- 
sion of resources from the State on programs that are in need of assistance and improve- 
ment."^^ As these examples illustrate, performance-based accountability carries with it large 
technical assistance responsibilities for states. 

FORMAL REQUIREMENTS OF PERKINS II FOR LOCAL PROGRAMS 

Congress recognizes the significance of LEAs fcr the success of vocational education. This is 
clear from the legislative history of Perkins II: "For the first time in federal vocational edu- 
cation history this landmark legislation sends all secondary education money dirsctly to local 
school districts or to area vocational schools serving such districts."^^ LEAs are vested with 
three major responsibilities: to provide service, to develop a local plan, and to evaluate per- 
formance and program improvement.^^ 

Provision of Sei^ces 

Congress provides a broad description of the role of LEAs: "the local provider is to serve the 
economically disadvantaged, individuals with disabilities, limited-English-proficient students 
and for programs to achieve sex equity [sic]."^"* The details of the programs that are to be 
provided by LEAs are spelled out in Perkins 11.^^ For example, services may include reme- 
dial courses, apprenticeship programs, supplementary services designed to meet the needs of 
special populations, and programs that integrate academic and vocational education.^^ 

Moreover, under Perkins II, Congress has limited the funds that can be spent for administra- 
tive tasks and has specified the minimum level of funds that must be distributed to LEAs. 
Together these actions ensure that the bulk of federal funds will be spent on providing direct 
services: 

The Committee recognizes that effective administration of vocational education programs can 
only be accomplished with fiscal support, but it is the Committee's clear intent that the primary 



S^lbid., Sec. 117(cXl). 

^^U.S. Code Congressional and Administrative News (1991), p. 1204. 
32lbid., p. 1183. 

^^The primary responsibilities of LEAs are specified in TITLE I— VOCATIONAL ASSISTANCE TO THE 
STATES, and TITLE II— BASIC STATE GRANTS FOR VOCATIONAL EDUCATION of Perkins II. LEAs have 
additional responsibilities for special programs (TITLE III) that we do not address in this report because of their 
more highly specialized nature. 

^Op. cit., p. 1201. 

^^See for example, Sees. 235(b) and (c) of Perkins II. 

^^Regulations governing the activities supported under TITLE II are included in 34 CFR Part 400, Subpart E. 
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use of the federal dollar should b that of direct services. In the Committee*s view, the five per- 
cent limitation on administration with the required match achieves that objective.^*^ 

Congress intends that federal funds be used to provide direct services. However, it is not 
clear that SEAs or LEAs have sufficient resources to undertake the roles assigned to them 
under Perkins II. 

Development of a Local Plan 

The second major responsibility of the LEAs is to develop a local plants based on a compre- 
hensive assessment of the needs of special populations.^^ The local plan must also describe 
the coordination of these programs with those supported under other related legislation, such 
as the JTPA. The local plan underlies the services that are the primary focus of LEAs,'*^ 

The requirement for LEAs to develop a local plan is not new. However, Perkins previously 
allowed the states greater latitude in defining the contents of local plans. In Perkins II, the 
state board still retains latitude in determining the requirements for the contents of local 
plans, but these requirements have been substantially expanded to include, for example, a 
description of program evaluation standards and a description of how the proposed programs 
will integrate academic and occupational disciplines. 

Several of these details have a direct bearing on accountability and serve to convey the clear 
federal intent that responsibility be driven down to the local level. For example, the local 
plan must take into account the occupational needs of the area it serves; must coordinate 
programs with local community-based organizations, in consultation with parents and stu- 
dents of special populations; and must describe the evaluation standards it will use to mea- 
sure the progress of its programs. Each of these requirements can be mapped directly onto 
the model of local accountability described in Section 2. 

Evaluation of Performance and Program Lnprovement 

The third major responsibility of the LEAs is performance evaluation and program improve- 
ment. This is an entirely new and important requirement initiated in Perkins II, and it 
underscores Congress' intent that accountability have a strong local component. There are 
four references to the responsibility of the LEAs in this regard in Perkins II; some have been 
mentioned previously, but all are listed here. The first of these simply requires that LEAs, in 
their local plan, "describe the program evaluation standards the applicant will use to mea- 
sure its progress."^^ Second, although the state is mandated to develop a system of standards 
and measures of performance, Perkins II provides for the final decision to be made at the 



^'U.S, Code Congressional and Adminisiraiiue News (1991), p. 1197. 

^^Perkins and Perkins II specify these requirements under the heading "Local Application." However, we refer 
to it as the "local plan," because the local application is primarily a description of programs and services that the 
LEA intends to provide, along with a description of how those programs will be administered and evaluated. Tlan" 
seems to us to convey more readily the intent and content of the local "application." 

^^Op, cit., p. 1203. 

^See Perkins II, Sec, 240, and 34 CFR Part 400 §403.190. for a description of what must be included in local 
plana. 

*^Perkins II, Sec. 240(6). 
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local level by allowing LEAs to make local modifications based on economic, geographic, or 
demographic factors.'^^ Third, LEAs are required to undertake an annual evaluation of 
programs conducted with assistance from Perkins 11.'*^ Fourth, if after one year, an LEA de- 
termines that it is not making substantial progress in meeting the standards and measures 
described above, it must develop a program improvement plan.^ 

The initial responsibility for judging whether an LEA is making "substantial progress" in 
meeting the standards and measures lies clearly with the LEAs. "If . , , a recipient deter- 
mines, through its annual evaluation, that it is not making substantial progress in meeting 
the standards and measures . . . the recipient shall develop a plan for program improvement 
for the succeeding school year.*"^ Yet LEAs do not have unconditional responsibility for ac- 
countability. Once an LEA has determined that progress toward achieving standards is not 
being made and has undertaken to implement a local improvement plan, future judgments as 
to whether substantial progress is being made are left to the state. According to the regula- 
tions: 

If, one year after an eligible recipient has implemented its program improvement plan described 
in §403.192, the state finds that the eligible recipient has not made sufficient progress in meeting 
the standards and measures developed as required by §§403,201 and 403.202, the state shall 
work jointly with the recipient and with teachers, parents, and students concerned with or af- 
fected by the program, to develop a joint plan for program improvement.^^ 

However, it would appear that LEAs can forestall any state intrusion. The potential exists 
for LEAs to modify state standards and measures to make them more lenient, and then to 
judge whether they continue to make substantial progress toward accomphshing them. Such 
a strategy would effectively preclude the need for developing a program improvement plan, 
would limit state interference in local administration, and would maintain local control over 
vocational programs.^*^ 

SUMMARY OF LEGISLATIVE EFFECTS ON THE PRACTICE OF 
LOCAL ACCOUNTABILITY 

In our judgment, the most important features of Perkins II, from the standpoint of local ac- 
countability in vocational education, are the provisions that: 

a. Place the primary responsibility for program evaluation and improvement squarely on the 
shoulders of local programs, while circumscribing the role of SEAs to monitoring and 
technical assistance. 

b. Define program success in terms of outcomes, and mandate that each state adopt its own 
common set of outcome-based standards and measures to judge program performance. 



'^^Ibid., Sec. 115(a). 
^Ibid., Sec. 117(a). 
*^Ibid., Sec. 117(b). 
^34 CFR Part 400 § 403.192. 
^34 CFR Part 400 § 403.204. 

^'^Thia appears to be a technical loophole whose impact will not be knov/n until the system has been in operation 
for a year or more. Conversations with state directors suggest that few LEAs are aware of this option at present. 
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It is also noteworthy that Perkins II defines accountability in terms of professional responsi- 
bility for program improvement, without implicit financial rewards or punishments. Taken 
together, these provisions place a greater burden on LEAs than they have carried in the past, 
while at the same time curtailing the potential for SEAs to provide assistance. 
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